Apparatus and method for detecting letter box, and MPEG decoding device having the same

ABSTRACT

In a device and a method for detecting a letter box for an MPEG decoder, the method includes performing processing area filtering for selecting a processing area of an image used to detect the letter box; performing intra-macroblock filtering for determining the letter box area based on a change level of pixels in macroblocks in one line of the image from the processing area; performing impulse data filtering for excluding the line being detected a high frequency component from the determined letter box area; performing inter-macroblock filtering for determining the letter box area based on a change level of lines between macroblocks of the image; performing inter-line filtering for determining a boundary of the letter box based on an average of the pixel values of the lines; and performing inter-picture filtering for outputting a boundary value of the letter box that has the highest frequency number as the boundary of the letter box in successive images.

RELATED APPLICATIONS

This application is a divisional application of U.S. patent applicationSer. No. 11/137,279, filed on May 25, 2005, which claims the benefit ofKorean patent application number 10-2004-0037813, filed on May 27, 2004,in the Korean Intellectual Property Office, the contents of whichapplications are incorporated herein in their entirety by reference.

BACKGROUND OF THE INVENTION

The present invention is concerned with MPEG decoding technology, andmore specifically relates to an apparatus and method for detecting aletter box for an MPEG decoder.

FIGS. 1A through 1C illustrate a letter box processing feature wherebyvideo signals of a 16:9 aspect ratio are converted into video signals ofa 4:3 aspect ratio in an encoder, and an complementary feature wherevideo signals of a 4:3 aspect ratio, containing a letter box, areconverted into video signals of a 16:9 aspect ratio.

Most existing televisions are configured for display in the 4:3 aspectratio. Accordingly, images for display that have the 16:9 aspect ratioas shown in FIG. 1A are first converted into images of 4:3 aspect ratioas shown in FIG. 1B, which is referred to as an MPEG encoding operation.In this case, in order to match the different aspect ratios of images toeach other, a standard of MPEG-2 referred to as Pan-Scan is used orimages are encoded by inserting invalid image data into upper and lowerareas of images, for example into letter box regions a and a′.

However, in the case where the image shown in FIG. 1B is decoded andinjected on a screen having a 16:9 or 4:3 aspect ratio, while the imagematches with the aspect ratio of the 4:3 screen, an image mismatchoccurs with the aspect ratio on the 16:9 screen, so that left and rightregions of the screen do not correspond to pixel values of decodingimage. Accordingly, a screen without left and right regions b and b′ isdisplayed or an image is injected thereon after interpolation, resultingin an unnatural image pattern. Also, since ineffective images appear atthe upper and lower regions, a and a′, or appear at the left and rightregions, b and b′, space on the screen display is wasted. In addition,for a television employing a CRT (Cathode-Ray Tube) as a monitor, a samepixel value is continuously output, which can cause screen damage.

In this case, when a decoded image is displayed on a screen, if images aand a′ inserted upon encoding are removed, and therefore an original16:9 image is injected on the 16:9 screen, it is possible to watch animage that has not yet been encoded in its original state. However,since it is impossible for the MPEG decoder to identify the size of theimage inserted upon encoding the MPEG, the resulting size of the imageis selected by a user in accordance with his own reference using apredetermined size. Furthermore, as the numerical value is incorrect,this technique is ineffective in image reproduction. Therefore, in orderto display a correct image in accordance with the aspect ratio of thescreen, it is important to correctly detect the size of the letter box aand a′ inserted into the upper and lower regions upon encoding.

SUMMARY OF THE INVENTION

The present invention is directed to an apparatus and method fordetecting a letter box capable of correctly displaying a decoded screenaccording to an aspect ratio by correctly detecting the size of theletter box that is inserted in an image, and an MPEG decoding deviceusing the same.

In one aspect, the present invention is directed to a method fordetecting a letter box, comprising the steps of performing processingarea filtering for selecting a processing area of an image used todetect the letter box; performing intra-macroblock filtering fordetermining the letter box area based on a change level of pixels inmacroblocks in one line of the image from the processing area;performing impulse data filtering for excluding the line being detecteda high frequency component from the determined letter box area;performing inter-macroblock filtering for determining the letter boxarea based on a change level of lines between macroblocks of the image;performing inter-line filtering for determining a boundary of the letterbox based on an average of the pixel values of the lines; and performinginter-picture filtering for outputting a boundary value of the letterbox that has the highest frequency number as the boundary of the letterbox in successive images.

In one embodiment, the method for detecting a letter box is performedafter decoding operations for the image have been performed.

In another embodiment, the method for detecting a letter box isperformed in units of a plurality of macroblocks included in the image.

In another embodiment, the processing area filtering step excludes aboundary part of the image where there may exist unusual pixels in theimage.

In another embodiment, the intra-macroblock filtering step includes thesub-steps of: calculating difference values between pixels in one linewith respect to a plurality of macroblocks of the image; calculatingpixel variations every one line with respect to the macroblocks on thebasis of the difference value; calculating a change level value of thepixels in one line of the image on the basis of the change level valueof the pixels in one line with respect to the macroblocks; comparing thechange level value of the image in one line of the image with a firstthreshold value; and determining that the corresponding line is notincluded in the letter box area when the change level value of the imagein one line of the image is higher than the first threshold value.

In another embodiment, the difference value between pixels correspondsto a difference value between pixels existing in positions separated bya desired distance on the same line of the macroblock.

In another embodiment, the impulse data filtering step includes thesub-steps of: comparing the difference values between pixels in one linecalculated in the intra-macroblock filtering step with a secondthreshold value; and determining that the corresponding line does notbelong to the letter box area when the difference value between thepixels is higher than the second threshold value.

In another embodiment, the inter-macroblock filtering step includes thesub-steps of: calculating the sum of pixels in one line with respect toeach of a plurality of macroblocks of the image; calculating a changelevel value in one line of each block by comparing an average of the sumof pixels in one line of the calculated macroblocks with an average ofthe sum of pixels in one line of adjacent macroblocks; calculating achange level value in one line of the image on the basis of the changelevel value of the macroblocks; comparing the change level value in oneline of the image with a third threshold value; and determining that thecorresponding line does not belong to the letter box area if the changelevel value in one line of the image is higher than the third thresholdvalue.

In another embodiment, the inter-line filtering step includes thesub-steps of: calculating an average of pixels included in the line;comparing an average value of pixels of the line with the average valueof pixels of the previous line; and determining that the correspondingline does not belong to the letter box area if a difference between theaverage values is higher than a desired threshold value.

In another aspect, the present invention is directed to an apparatus fordetecting a letter box, including a processing area filter for selectinga processing area of an image used to detect the letter box; anintra-macroblock filter for calculating a change level of pixels in oneline of the image from the processing area in the macroblock, anddetermining whether the corresponding line belongs to the letter boxarea by comparing the calculated change level of the pixel in one linewith a first threshold value; an impulse data filter for detecting ahigh frequency component from the line determined to be included in theletter box in the intra-macroblock filter; an inter-macroblock filterfor calculating a change level of an image with respect to the line thatdoes not have the high frequency component between macroblocks, anddetermining whether the corresponding line belongs to a letter box areaby comparing the change level of the image in one line with a thirdthreshold value; an inter-line filter for determining a boundary of theletter box by comparing the line determined to be included in the letterbox in the inter-macroblock filter with an average of the pixel valuesof adjacent lines; and an inter-picture filter for repeatedly performingoperations of the processing area filter, the intra-macroblock filter,the impulse data filter, the inter-macroblock filter, and the inter-linefilter as many times as the number of the image previously set, andoutputting a boundary value of the letter box that has the highestfrequency number among the performed results.

In one embodiment, the letter-box detection is performed after all ofdecoding operations for the image have been performed.

In another embodiment, the filtering is performed in units of aplurality of macroblocks included in the image.

In another embodiment, the processing area filter excludes a boundarypart of the image where there may exist unusual pixels in the image.

In another embodiment, the intra-macroblock filter calculates differencevalues between pixels in one line with respect to a plurality ofmacroblocks that construct the image, compares the change level value ofpixels in one line of the calculated macroblocks on the basis of thedifference value with the first threshold value, and determines that thecorresponding line does not belong to the letter box area if the changelevel value of pixels in one line of the image is higher than the firstthreshold value.

In another embodiment, the difference value between pixels is adifference value between pixels existing in positions separated by adesired distance on the same line of the macroblock.

In another embodiment, the impulse data filter compares the differencevalue between the pixels with the second threshold value, and determinesthat the corresponding line does not belong to the letter box area ifthe difference value between the pixels is higher than the secondthreshold value.

In another embodiment, the inter-macroblock filter calculates the changelevel value of each macroblock in one line by comparing an average of asum of pixels of each of a plurality of macroblocks of the image and asum of pixels in one line of adjacent macroblocks, compares the changelevel value in one line of the image calculated on the basis of thechange level value in one line with the third threshold value, anddetermines that the corresponding line does not belong to the letter boxarea if the change level value of the image is higher than the thirdthreshold value.

In another embodiment, the inter-line filter compares the average valueof the pixel of the line with the average value of the pixel of theprevious line, and determines that the corresponding line does notbelong to the letter box area if the difference between the averagevalues is higher than a fourth threshold value.

In another aspect, the present invention is directed to an MPEG decodingdevice, comprising a decoding unit for decoding an image including aletter box; a letter-box detection unit for extracting a processing areaof the decoded image, and determining whether the corresponding linebelongs to a letter box area by calculating a change level of pixels ofthe image and a change level of a line of the image in unit of aplurality of macroblocks included in the extracted processing area; andan image display control unit for responding to the letter-box detectionresult, and controlling and displaying the size of the image; whereinthe letter-box detection unit performs a letter-box detection operationas many times as the number of the detection previously set and outputsa letter box boundary value having the highest frequency number.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of theinvention will be apparent from the more particular description ofpreferred embodiments of the invention, as illustrated in theaccompanying drawings in which like reference characters refer to thesame parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the invention. In the drawings:

FIGS. 1A through 1C are diagrams illustrating a letter box processingfeature where video signals of 16:9 aspect ratio are converted intovideo signals of 4:3 aspect ratio in an encoder, and an exemplaryfeature where video signals of 4:3 aspect ratio, containing a letterbox, are converted into video signals of 16:9 aspect ratio;

FIG. 2 is a block diagram of an MPEG decoding device in accordance withan embodiment of the present invention;

FIG. 3 is a block diagram of a letter-box detector in accordance with anembodiment of the present invention;

FIG. 4 is a diagram illustrating an image construction to be used todetect a letter box and a filtering procedure performed by theprocessing area filter shown in FIG. 3;

FIG. 5 is a diagram illustrating a construction of the macroblock shownin FIG. 4 and a filtering procedure by the intra-macroblock filter shownin FIG. 3;

FIG. 6 is a diagram explaining a filtering procedure of theinter-macroblock filter shown in FIG. 3; and

FIGS. 7A and 7B are flow charts that illustrate a method for detecting aletter box in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will be described belowin more detail with reference to the accompanying drawings. The presentinvention may, however, be embodied in different forms and should not beconstructed as limited to the embodiments set forth herein. Rather,these embodiments are provided so that this disclosure will be thoroughand complete, and will fully convey the scope of the invention to thoseskilled in the art. Like numerals refer to like elements throughout thespecification.

Hereinafter, an exemplary embodiment of the present invention will bedescribed in conjunction with the accompanying drawings.

According to a novel apparatus and method for detecting a letter boxletter box in accordance with the present invention, an MPEG decodingdevice performs a decoding of an image, and then detects the size of theletter box inserted into the image through a six step filtering processwithout affecting the result of the decoding and without requiringadditional memory access operations. As a result, it is possible tocorrectly display the decoded image according to the aspect ratio.

FIG. 2 is a block diagram illustrating an MPEG decoding device 100 inaccordance with a preferred embodiment of the present invention.

Referring to FIG. 2, the MPEG decoding device 100 in accordance with thepresent invention decodes an encoded bit stream STREAM_ENC, detects thesize of the letter box inserted into the image, and adjusts and outputsthe decoded image to be matched with the screen aspect ratio of adisplay unit 90. As a result, the decoded image can be correctlydisplayed according to the screen aspect ratio of the display unit 90.

In order to accomplish the correct display, the MPEG decoding device 100includes an external memory 10 that is used as a video memory forstoring image data; a bus 20 connected to each of logic units includedin the MPEG decoding device 100 and used as a path for transceivingsignal and data information; an MPEG decoder 30 for decoding the encodedMPEG bit stream STREAM_ENC; a system control unit 70 for controlling allof operations of the MPEG decoding device 100; and a display controlunit 80 for adjusting and outputting the decoded result to be matchedwith the screen aspect ratio of the display unit 90.

The MPEG decoder 30 is composed of a video stream interface unit 40 forinterfacing to the encoded bit stream STREAM_ENC; an MPEG decoding unit50 for performing a decoding operation of the encoded bit streamSTREAM_ENC; and a letter-box detection unit 60 for detecting a letterbox inserted into the image upon encoding, wherein the MPEG decoder 30performs a decoding operation of image, and an operation of detectingthe size of the letter box inserted into the image.

After the MPEG decoding unit 50 performs a series of decoding procedures(for example, an Inverse Discrete Cosine Transform: IDCT), and a motioncompensation procedure, the letter-box detection unit 60 reads out dataused to store the decoded data in the external memory and detects thesize of the letter box existing in the image. The letter box informationdetected in the letter-box detection unit 60 is transmitted to thedisplay control unit 80 through the system control unit 70. The displaycontrol unit 80 responds to a control signal and the letter boxinformation input from the system control unit 70 and adjusts andoutputs the decoded image to be matched with the screen aspect ratio ofthe display unit 90.

In accordance with the present invention, each of the filters of theletter-box detection unit 60 is constructed of dedicated hardware inorder to optimize its speed, data storage space and the like. However,it is possible to embody a letter-box detection function on a computerreadable recording medium as computer readable code. The computerreadable recording medium includes all kinds of recording apparatuses inwhich data that can be read by a computer system are stored, andexamples of the computer readable recording medium include a ROM, a RAM,a CD-ROM, a magnetic tape, a floppy disk, an optical data storageapparatus, and the like. Also, the computer readable recording mediumcan be distributed by the computer system connected by a network, andstored and implemented as the computer readable code using adistribution scheme.

FIG. 3 is a block diagram of the letter-box detection unit 60 inaccordance with an embodiment of the present invention shown in FIG. 2.

Referring to FIG. 3, the letter-box detection unit 60 in accordance withthe present invention includes a processing area filter (PAF) 61, anintra-macroblock filter (INT_MF) 62, an impulse data filter (IDF) 63, aninter-macroblock filter (IMF) 64, an inter-line filter (ILF) 65, andinter-picture filter (IPF) 66.

Conventional types of letter-box detection units read out pixels inunits of lines from the external memory 10 and process pixels in unitsof lines. However, since the MPEG operation performs lossy compressionin units of macroblocks, a changing level of the pixel value is changedin units of macroblocks. Accordingly, the present invention increasesprecision for detecting a letter box by performing five steps offiltering in unit of the macroblocks in consideration of such an MPEGproperty. Particularly, since the letter-box detection unit 60 inaccordance with the present invention detects the letter box bysearching for a signal that appears upon storing the final pixel valueoccurring when the MPEG decoder finishes the decoding operation, theoperation of the unit 60 does not adversely affect system performance.Operations of each filter of the letter-box detection unit 60 havingsuch a construction will be described as follows.

FIG. 4 is a view showing a construction of an image to be used to detectthe letter box and a filtering procedure of the processing area filter61 shown in FIG. 3.

Referring to FIG. 4, the processing area filter 61 performs a functionto exclude an edge part of the image where there abnormal pixels of thedecoded image 200 may exist and to select a processing area 220 wherestable data exist. As the data are processed in several steps to encodeand decode the image, there may be abnormal pixels in the edge area 210of each image. Accordingly, the present invention increases theprecision for detecting a letter box by excluding the edge area 210 ofthe image where the abnormal pixels may exist from the letter box usingthe processing area filter 61.

FIG. 5 illustrates a construction of the macroblock shown in FIG. 4 andfiltering procedures of the intra-macroblock filter 62 shown in FIG. 3.

Referring to FIGS. 4 and 5, the intra-macroblock filter 62 calculatesdifferences between pixels d00, d01, . . . , df6, df7 in one line ofeach macroblock 221 included in the effective data area selected throughthe processing area filter 61, and measures a change level of pixelNO_MB_LB in one line of the macroblock on the basis of the calculateddifferences between pixels d00, d01, . . . , df6, df7. After theNO_MB_LB value in one line of the image is obtained by summing NO_MB_LBvalues in one line of several macroblocks, it is determined whether anarbitrary line is included in the letter box by comparing the NO_MB_LBin one line of the image with a desired threshold TH_NUM_INTRA_LINE_MB.A filtering procedure performed in the intra-macroblock filter 62 isdescribed in detail as follows.

The image 220 filtered by the processing area filter 61 is constructedof a plurality of macroblocks 221, 222, 223, . . . as shown in FIG. 4,and each of the plurality of macroblocks 221, 22, 223, . . . isconstructed of 16*16 pixels as shown in FIG. 5.

First, the intra-macroblock filter 62 is used to calculate thedifference between pixels d00, d01, . . . df6, df7 in one line of eachmacroblock 221. At this time, the calculated difference between pixelsd00, d01, . . . df6, df7 and corresponding relationships between pixelsis as follows.d00=|p00−p08|, d01=|p01−p09|d02=|p02−p0a|, d03=|p03−p0b|d04=|p04−p0c|, d05=|p05−p0d|d06=|p06−p0e|, d07=|p07−p0f|d10=|p10−p18|, d11=|p11−p19|d12=|p12−p1a|, d13=|p13−p1b|d14=|p14−p1c|, d15=|p15−p1d|d16=|p16−p1e|, d17=|p17−p1f|df0=|pf0−pf8|, df1=|pf1−pf9|df2=|pf2−pfa|, df3=|pf3−pfb|df4=|pf4−pfc|, df5=|pf5−pfd|df6=|pf6−pfe|, df7=|pf7−pff|  [Equation 1]

Here, the reason why obtaining the difference between a pixel and apixel separated by 8 pixels from the first one instead of obtaining thedifference between adjacent two pixels is that the inverse discretecosine transform (IDCT) is performed in units of 8×8, and obtaining thedifference (that is, a change) between a pixel in an arbitrary block anda pixel belonging to another block is available to more precisely detecta letter box.

After differences between pixels in one line d00, d01, . . . df6, df7 ofeach of macroblock 221 are calculated in the Equation 1, the number ofdifferences between pixels in one line d00, d01 m, . . . df6, df7 higherthan a desired threshold value TH_INTRA_MB is obtained in Equation 2(NUM[(From d00 to d07)>TH_INTRA_MB]), and it is determined whether theobtained number is higher than the desired threshold valueTH_NUM_INTRA_MB. If it is determined that the obtained number is higherthan the threshold value as a result of the determination, a changelevel of pixel of the macroblock NO_MB_LB in one line is calculated byincreasing the count value one by one. Such a calculation is performedin each line of each macroblock.If NUMI[(From d00 to d07)>TH _(—) INTRA _(—) MB]>TH _(—) NUM _(—) INTRA_(—) MB_NO_(—) MB _(—) LB=NO_(—) MB _(—) LB+1  [Equation 2]

Here, as the change level value of the pixel of the macroblock NO_MB_LBin one line increases, the probability that the corresponding line isnot a letter box becomes higher.

After both procedures of Equations 1 and 2 and are performed in eachline of each macroblock, the change level between pixels NO_MB_LB in oneline for one row line of the image formed of a plurality of macroblocksis calculated. The change level between pixels NO_MB_LB in each line ofan image is calculated by summing change levels between pixels in eachline of a plurality of macroblocks 221, 222, 223, . . . that arearranged in a horizontal direction (that is, in a row direction) of animage.

After the change level between pixels NO_MB_LB in one line of an imageis calculated, a change level between pixels NO_MB_LB in one line of animage is compared with a desired threshold value TH_NUM_INTRA_LINE_MBand it is determined whether the corresponding line belongs to a letterbox area.NO_(—) MB _(—) LB in one line>TH _(—) NUM _(—) INTR_LINE_(—)MB  [Equation 3]

When the change level between pixels NO_MB_LB in one line of an image ishigher than a desired threshold value TH_NUM_INTRA_LINE_MB as a resultof the comparison in Equation 3, it is determined that the correspondingline is not a letter box (that is, determined as an active area). Also,when the change level between pixels NO_MB_LB in one line of an image islower than the desired threshold value TH_NUM_INTRA_LINE_MB as a resultof the comparison in Equation 3, the corresponding line can be anindication of the presence of a letter box.

When filtering of the intra-macroblock filter 62 is completed, filteringfor the pixel selected as a letter box is performed by an impulse datafilter 63. The impulse data filter 63 operates to supplement theintra-macroblock filter 62. That is, the intra-macroblock filter 62determines a letter box by measuring the number of changed pixels in themacroblock, and the impulse data filter 63 determines a letter box bymeasuring the change in the level of the pixel. The impulse data filter63 detects a high frequency component in an image, where it isdetermined that a line having at least one of pixels having a highfrequency component is not a letter box area.

This is because an MPEG image is processed in a pre-processing proceduresuch as a low pass filter LPF and the like before it is encoded and mostof impulse noise is thus removed. Also, since a noise characteristic ofthe noise that occurs due to loss compression in the MPEG removes a highfrequency component between pixels, if there exists at least one ofpixels that have a severe difference in a portion corresponding to theletter box in the intra-macroblock filter 62, the probability is higherthat the line is a letter box. Accordingly, the impulse data filter 63determines whether the differences between pixels d00, d01, . . . d0 fobtained by the Equation 1 are higher than a desired threshold valueTH_MAX_DIFF_PIX (that is, whether there exists a high frequencycomponent). If the differences between pixels d00, d01, . . . d0 f arehigher than the desired threshold value TH_MAX_DIFF_PIX (that is, ifthere exists the high frequency component) as a result of thedetermination, it is determined that the corresponding line is not aletter box. The filtering procedure of the impulse data filter 63 isalso performed for all lines included in each macroblock.

Following filtering by the impulse data filter 63, the letter-boxdetection unit 60 in accordance with the present invention performsfiltering by the inter-macroblock filter 64.

FIG. 6 is a view explaining a filtering procedure of theinter-macroblock filter 64 shown in FIG. 3.

Referring to FIGS. 4 and 6, the inter-macroblock filter 64 calculatesthe sum A0-A15, B0-B15, C0-C15, . . . of pixels in one line of each ofmacroblocks 221, 222, 223, . . . , and change levels NO_MB_LB1 in oneline of each macroblock are obtained on the basis of the calculated sumA0-A15, B0-B15, C0-C15, . . . of pixels in one line of each of themacroblocks, and an average of adjacent macroblocks. Also, a changelevel NO_MB_LB1 in one line of an image is obtained by summing changelevels NO_MB_LB1 in one line of a plurality of macroblocks, and then itis determined whether the corresponding line belongs to a letter boxarea by comparing the change level of macroblocks NO_MB_LB1 in one linewith a desired threshold TH_NUM_INTER_MB. The filtering procedureperformed in an inter-macroblock filter 64 is as follows.

An image from which a letter box is detected is constructed of aplurality of macroblocks 221, 222, 223, . . . and each of macroblocks221, 222, 223, . . . is constructed of 16 lines that each include 16pixels. First, the inter-macroblock filter 64 calculates the sum A0-A15,B0-B15, C0-C15, . . . of pixels in one line of the macroblocks 221, 222,223, . . . . Here, A0 indicates the sum of pixels of a first line of afirst macroblock 221 (that is, A0=p00+p01+p0 e+p0 f), and B0 indicatesthe sum of pixels of the first line of the second macroblock 222. Also,the summation of lines of the macroblock is performed in eachmacroblock.

After the sums A0-A15, B0-B15, C0-C15, . . . of pixels in one line ofeach macroblock are calculated, the difference between calculated sum ofpixels in one line of one macroblock and an average value of surroundingmacroblocks adjacent to the macroblock is obtained, and the differenceis compared with a desired threshold value TH_INTER_MB and then thechange level in one line between the macroblocks is obtained as shown inEquation 4.If |AVE(A0, B0 . . . )−C0|>TH _(—) INTER _(—) MBNO_(—) MB _(—) LB1=NO_(—) MB _(—) LB1+1  [Equation 4]

Here, as the change level value NO_MB_LB1 in each line between themacroblocks increases, the probability that the corresponding line isnot a letter box becomes higher. The calculation and comparison for thechange level NO_MB_LB1 in one line of the macroblock are performed withrespect to each of all macroblocks.

At this time, if it is determined that the change level in one line ofeach macroblock is higher than a desired threshold value TH_INTER_MB,this means that the probability that the corresponding line of thecorresponding macroblock is not a letter box becomes higher.Accordingly, when the change in one line of one macroblock is comparedwith that of a surrounding macroblock, the comparison is performedwithout including the corresponding line of the correspondingmacroblock. For example, in the case where the difference between a sumC0 of the first line of third macroblock 223 and an average value of thesums A0 and B0 of first lines of adjacent macroblocks 221 and 222 ishigher than the desired threshold value TH_INTER_MB, the sum C0 of thefirst line of the third macroblock 223 is not used when the comparisonand sum operation are performed with respect to a first line of thefourth macroblock. At this time, the sum of the pixels of the first lineof the fourth macroblock is compared with the average value of the sumsA0 and B0 of the first line of the first and second macroblocks 221 and222 excluding the sum C0 of the first line of the third macroblock 223.

When the procedure of Equation 4 is completed in one line of eachmacroblock, the change level between macroblocks NO_MB_LB1 in one lineof an image formed on a plurality of macroblocks is calculated. Thechange level between macroblocks NO_MB_LB1 in one line for an image iscalculated as a sum of change NO_MB_LB1 in one line of the plurality ofmacroblocks 221, 222, 223, . . . that are arranged in a width direction(that is, in a row direction) of the image.

Subsequently, the change level between macroblocks NO_MB_LB1 in one linefor an image is compared with a desired threshold value TH_NUM_INTER_MBand it is determined whether the corresponding line belongs to a letterbox area.NO_(—) MB _(—) LB1 in one line>TH _(—) NUM _(—) INTER _(—) MB  [Equation5]

As a result of the comparison in Equation 5, when the change levelbetween macroblocks NO_MB_LB1 in one line for an image is higher thanthe desired threshold value TH_NUM_INTER_MB, it is determined that thecorresponding line is not a letter box (that is, the line is an activearea). As a result of comparison result of the Equation 5, when thechange level between pixels NO_MB_LB1 in one line for an image becomeslower than the desired threshold value TH_NUM_INTER_MB, this indicatesthat the corresponding line can be a letter box.

Returning to FIG. 3, the inter-line filter 65 next searches for anaverage of pixel values of adjacent lines in an image and determines aboundary of the letter box in accordance with a characteristic that theletter box area is constructed of lines in which a desired interval hascontinuous values.If |AVE(current line)−AVE(previous line)|>TH_LINE_BY_LINE  [Equation 6]

In Equation 6, when the difference between the average value of pixelsof a current line and that of pixels of a previous line is greater thana desired threshold value TH_LINE_BY_LINE, it is determined that thecurrent line belongs to an area (that is, an active area) whose line isnot continuous to the current line. When the difference between theaverage value of pixels of current line and that of pixels of previousline is lower than a desired threshold value TH_LINE_BY_LINE, it isdetermined that the current line corresponds to a letter box. At thistime, the letter box located on the upper part of the image is searchedfor whether the continuation is present at the first line, and theletter box located on the lower part of the image is searched forwhether the continuation is present at the last line, so that theposition of each letter box for each image is determined.

When a letter box for an image is detected using the series of filteringprocedures described above, the inter-picture filter 66 determines thenumber of images pic_RANGE from which the letter boxes are detected.When the number of the images whose letter box is detected PIC_RANGE isless than a desired number NUM_PIC_LBD, the series of filteringprocedures described above are repeatedly performed until the number ofimages whose letter boxes are detected PIC_RANGE becomes a desirednumber NUM_PIC_LBD. When the number of the images whose letter box isdetected PIC_RANGE is higher than a desired number NUM_PIC_LBD, aninterrupt signal is generated and the size corresponding to the letterbox having the highest frequency number among the letter-box detectionresults performed up to that point in time is output.

Returning to FIG. 2, if the letter-box detection result in each image isoutput to a system control unit 70 of the MPEG decoding device 100, thesystem control unit 70 can be overloaded. Accordingly, the inter-picturefilter 66 (see FIG. 3) sets a desired number of the letter-boxdetection, and the letter-box detection result is not output to thesystem control unit 70 of the MPEG decoding device 100 until the letterbox is detected as many times as previously set.

FIGS. 7A and 7B are flow diagrams that a method for detecting a letterbox in accordance with an embodiment of the present invention.

Referring to FIGS. 7A and 7B, the method for detecting a letter box inaccordance with the present invention, firstly, performs a processingarea filtering PAF operation (step 610). The processing area filteringPAF procedure includes the step of receiving an image in which adecoding operation has been completed (step 612), and the step ofremoving a boundary part of the image and selecting a processing area ofthe image (step 614).

Subsequently, the intra-macroblock filtering is performed where a changelevel of the pixel NO_MB_LB in one line is obtained from the differencebetween the pixels calculated in one line of each macroblock and it isdetermined whether the corresponding line belongs to the letter box area(step 620). The intra-macroblock filtering procedure (step 620)calculates difference values between the calculated pixels d00, d01, . .. , df6, df7 in one line of the macroblock (step 622), and measures thechange level between pixels in one line on the basis of the differencevalues between the calculated pixels d00, d01, . . . , df6, df7 (step624). The change level between pixels in one line measured in the step624 includes the change level of the pixel in one line of the macroblockNO_MB_LB and the change level of the pixel NO_MB_LB in one line of theimage. The change level of the pixel NO_MB_LB in one line of the imageis calculated as a sum of the change levels between pixels NO_MB_LB inone line of a plurality of macroblocks arranged in the width direction(that is, in a row direction) of the image. After the change levelbetween pixels NO_MB_LB in one line of the image is measured in the step624, the intra-macroblock filtering procedure (step 620) determineswhether the change level between pixels in one line of the measuredimage is higher than the first threshold value TH_NUM_INTRA_LINE_MB, andthen whether the corresponding line is a letter box area (step 626).That is, if the change level between pixels in one line of the measuredimage is greater than the first threshold value TH_NUM_INTRA_LINE_MB asa result of the determination in the step 626, it is determined that thecorresponding line is not a letter box. Also, if the change levelbetween pixels in one line of the measured image is less than the firstthreshold value TH_NUM_INTRA_LINE_MB, the corresponding line can be aletter box.

After the intra-macroblock filtering INT_MF is performed in step 620, animpulse data filtering IDF to detect the magnitude of a high frequencycomponent existing in the image is performed (step 630). It isdetermined whether difference values between pixels calculated in step622 d00, d01, . . . df6, df7 are higher than the second threshold valueTH_MAX_DIFF_PIX in the step 630, and then it is determined whether thecorresponding line belongs to the letter box area. That is, if thedifferences between pixels calculated in one line of the macroblock d00,d01, . . . , df6, df7 is higher than the second threshold valueTH_MAX_DIFF_PIX, it is determined that the corresponding line is not aletter box area (that is, it is determined as an active area). Also, ifit is determined that the differences between pixels d00, d01, . . . ,df6, df7 are lower than the second threshold value TH_MAX_DIFF_PIX, thecorresponding line can belong to a letter box area.

Subsequently, the change level of macroblocks NO_MB_LB1 in one line ofthe image is calculated on the basis of the average of the sum of pixelsin one line included in macroblocks and the sum of pixels in one line ofadjacent macroblocks, and an inter-macroblock filtering IMF is performedto determine whether the corresponding line belongs to the letter boxarea (step 640).

In the inter-macroblock filtering step (step 640), sums of pixels in oneline for each macroblock A0-A15, B0-B15, C0-C15, . . . are calculated(step 642). A change level of each macroblock NO_MB_LB1 in one line isobtained by comparing the sums of pixels of the macroblock in one lineA0-A15, B0-B15, C0-C15, . . . with an average of adjacent macroblocks,and then a change level of macroblocks NO_MB_LB1 in one line of an imageby summing the change level of each macroblock in one line and a changelevel of a plurality of macroblocks in one line NO_MB_LB1 (step 644).Subsequently, it is determined whether the corresponding line belongs toa letter box area by comparing the change level of macroblocks NO_MB_LB1in one line of the calculated image in the step 644 with the thirdthreshold TH_NUM_INTER_MB (step 646). If the change level of themacroblocks NO_MB_LB1 in one line of the image is higher than the thirdthreshold TH_NUM_INTER_MB as a result of the determination in the step646, it is determined that the corresponding line does not belong to aletter box area (that is, the corresponding line is determined to be anactive area). If the change level of the macroblocks NO_MB_LB1 in oneline of the image is lower than the third threshold TH_NUM_INTER_MB, thecorresponding line can belong to a letter box area.

Following the inter-macroblock filtering of step 640, an inter-linefiltering ILF is performed where an average of pixel values of adjacentlines of the image is searched and then a boundary of continuous letterboxes is determined (step 650).

In the inter-line filtering step (step 650), an average value of pixelsin one line of the image is calculated (step 652), and it is determinedwhether a difference between an average value of pixels of a currentline and that of the previous line is higher than a desired thresholdvalue TH_LINE_BY_LINE so that it is determined whether the line iscontinuous (step 654). For example, if the difference between theaverage value of pixels of the current line and that of the previousline is higher than the desired threshold value TH_LINE_BY_LINE, it isdetermined that the current line belongs to an area different from thearea to which the previous line belongs. Also, if the difference betweenthe average value of pixels of the current line and that of the previousline is lower than the desired threshold value TH_LINE_BY_LINE, it isdetermined that the current line belongs to a letter box area. And then,a letter box size of the current image is determined according to theresult determined in the step 654 (step 656).

As described above, if a letter box size for one image is determinedaccording to a series of filtering procedures performed in the steps 610to 650, an inter-picture filtering IPF is performed where a letter-boxdetection is repeatedly performed as many times as previously set (step660).

In the inter-picture filtering step (step 660), the number of imagesPIC_RANGE in which steps 610 to 650 are performed is determined (step662), the determined number of images are searched and it is confirmedwhether the number of images whose letter boxes are detected is higherthan a fourth threshold NUM_PIC_LBD (that is, it is confirmed whetherthe letter box is detected as many as previously set (step 664). If thenumber of the images whose letter boxes are detected is higher than thefourth threshold NUM_PIC_LBD as a result of the determination in step664, the letter box size having the highest frequency number among theletter-box detection results performed until then is output (step 666).

According to the present invention, if a decoding of an image isperformed in an MPEG decoding apparatus, it is possible to performfiltering units of macroblocks and then detect a letter box sizeinserted in the image. As a result, it is possible to precisely displaythe decoded image according to the aspect ratio of screen.

While this invention has been particularly shown and described withreferences to preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade herein without departing from the spirit and scope of the inventionas defined by the appended claims.

1. A method for detecting a letter box, comprising the steps of:performing processing area filtering for selecting a processing area ofan image used to detect the letter box; performing intra-macroblockfiltering for determining the letter box area based on a change level ofpixels in macroblocks in one line of the image from the processing area;performing impulse data filtering for excluding the line being detecteda high frequency component from the determined letter box area;performing inter-macroblock filtering for determining the letter boxarea based on a change level of lines between macroblocks of the image;performing inter-line filtering for determining a boundary of the letterbox based on an average of the pixel values of the lines; and performinginter-picture filtering for outputting a boundary value of the letterbox that has the highest frequency number as the boundary of the letterbox in successive images.
 2. The method as set forth in claim 1, whereinthe method for detecting a letter box is performed after decodingoperations for the image have been performed.
 3. The method as set forthin claim 1, wherein the method for detecting a letter box is performedin units of a plurality of macroblocks included in the image.
 4. Themethod as set forth in claim 1, wherein the processing area filteringstep excludes a boundary part of the image where there may exist unusualpixels in the image.
 5. The method as set forth in claim 1, wherein theintra-macroblock filtering step includes the sub-steps of: calculatingdifference values between pixels in one line with respect to a pluralityof macroblocks of the image; calculating pixel variations every one linewith respect to the macroblocks on the basis of the difference value;calculating a change level value of the pixels in one line of the imageon the basis of the change level value of the pixels in one line withrespect to the macroblocks; comparing the change level value of theimage in one line of the image with a first threshold value; anddetermining that the corresponding line is not included in the letterbox area when the change level value of the image in one line of theimage is higher than the first threshold value.
 6. The method as setforth in claim 5, wherein the difference value between pixelscorresponds to a difference value between pixels existing in positionsseparated by a desired distance on the same line of the macroblock. 7.The method as set forth in claim 5, wherein the impulse data filteringstep includes the sub-steps of: comparing the difference values betweenpixels in one line calculated in the intra-macroblock filtering stepwith a second threshold value; and determining that the correspondingline does not belong to the letter box area when the difference valuebetween the pixels is higher than the second threshold value.
 8. Themethod as set forth in claim 1, wherein the inter-macroblock filteringstep includes the sub-steps of: calculating the sum of pixels in oneline with respect to each of a plurality of macroblocks of the image;calculating a change level value in one line of each block by comparingan average of the sum of pixels in one line of the calculatedmacroblocks with an average of the sum of pixels in one line of adjacentmacroblocks; calculating a change level value in one line of the imageon the basis of the change level value of the macroblocks; comparing thechange level value in one line of the image with a third thresholdvalue; and determining that the corresponding line does not belong tothe letter box area if the change level value in one line of the imageis higher than the third threshold value.
 9. The method as set forth inclaim 1, wherein the inter-line filtering step includes the sub-stepsof: calculating an average of pixels included in the line; comparing anaverage value of pixels of the line with the average value of pixels ofthe previous line; and determining that the corresponding line does notbelong to the letter box area if a difference between the average valuesis higher than a desired threshold value.